CCNTCR  for  naval  ANALYSES  ARLINGTON  VA  r/fl  -yo 

INFORMATION  ^'’OCESSINO  ANO  OUTCOME  FORECASTING  FOR  MULTILATERAL-ETC  (U) 

CNA-PROrcSSlONAL  PAPCft-lG  »ji 


JfflCflLECOPK  ADA042322 


INFORMATION  PROCESSING  AND  OUTCOME  FORECASTING  FOR 
MULTILATERAL  NEGOTIATIONS:  TESTING  ONE  APPROACH , 


The  ideas  expressed  in  this  paper  are  those  of  the 
author.  The  paper  does  not  necessarily  represent 
the  views  of  either  the  Center  for  Naval  Analyses 
or  the  Department  of  Defense. 


CENTER  FOR  NAVAL  ANALYSES  / 

1401  Wilson  Boulevard 
Arlington,  Virginia  22209 


Prepared  for  presentation  to  the  18th  Annual 
Convention  of  the  Intenutional  Studies  Associa- 
tion, Chase-Park  Plaza  Hotel,  St.  Louis,  Missouri, 
March  16-20, 1977. 


i 


INFORMATION  PROCESSING  AND  OUTCOME  FORECASTING  FOR 
MULTILATERAL  NEGOTIATIONS:  TESTING  ONE  APPROACH 


by  William  J.  Durch 
Center  for  Naval  Analyses 


The  views  expressed  in  this  paper  are  those  of  the 
author  and  do  not  necessarily  represent  those  of 
the  Center  for  Naval  Analyses  or  the  Department  of 
the  Navy. 

Prepared  for  presentation  to  the  18th  Annual  Conven- 
tion of  the  International  Studies  Association,  Chase- 
Park  Plaza  Hotel,  St.  Louis,  Missouri,  March  16-20, 
1977. 


CLEARED  FOR  PUBLIC  RELEASE 
Distribution  Unlimited 


ACKNOWLEDGEMENTS 


I would  like  to  thank  Robert  L.  Friedheim  and 
Joseph  B.  Kadane  for  their  suggestions  regarding  the 
research  design  of  the  reliability  tests  presented  here- 
in; Karen  W.  Goudreau  for  her  help  in  setting  up  the 
analytic  framework  and  programming  for  the  tests;  and 
Robert  B.  Mahoney  for  his  considerable  methodological 
advice  and  review  of  an  early  draft  of  the  paper.  The 
responsibility  for  opinions,  errors  and  omissions,  how- 
ever, is  my  own. 


TABLE  OF  CONTENTS 


Page 

Introduction 

Law  of  the  Sea  Project  Methodology 3 

Coding  Reliability 14 

Policy  Scaling  Reliability  26 

A Validity  Check:  LOS  and  the  SNT 41 

Sununary  and  Conclusions 44 


ii 


INTRODUCTION 

Multilateral  negotiations  generate  enormous  amounts  of 
written  materials,  which  resemble  useful  information  in  about 
the  same  way  that  iron  ore’  resembles  steel.  The  processing  of 
those  materials  into  useful  information  involves  extraction  of 
data,  its  storage  and  reduction  to  summary  indicators  of  trends 
in  bargaining,  participants'  positions  on  key  issues,  and  so  forth. 
The  purpose  of  this  paper  is  to  evaluate  a data  base 

management  system  for  multilateral  negotiations^ developed  at  the 
Center  for  Naval  Analyses  by  Robert  L.  Friedheim  and  Joseph  B. 
Kadane.^  Coupled  with  forecasting  models  developed  by  Kadane, 
the  system  was  used  by  CNA's  Law  of  the  Sea  (LOS)  Project  to 
provide  analytic  support  to  U.S.  Navy  and  other  U.S.  government 
negotiators  at  the  United  Nations  Law  of  the  Sea  negotiations, 
in  both  the  Seabed  Committee  and  later,  the  LOS  Conference. 

The  paper  will  principally  report  the  results  of  two 
tests,  designed  by  the  author,  to  determine  the  reliability  of 
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^Robert  L.  Friedheim  is  now  associate  director  for  marine 
policy  studies  at  the  Institute  for  Marine  and  Coastal  Studies, 
University  of  Southern  California.  Joseph  B.  Kadane  is  chairman 
of  the  statistics  department  at  Carnegie-Mellon  University.  The 
author  was  associated  with  the  LOS  project  from  the  end  of  1973 
through  mid-1976.  The  tests  reported  here  were  given  in  the 
spring  of  1976.  For  an  early  exposition  of  the  LOS  project's 
methodology,  see  Friedheim  and  Kadane,  "Ocean  Science  in  the  UN 
Political  Arena,"  Journal  of  Maritime  Law  and  Commerce  3, 

April  1972:  473-5021 


the  project's  thematic  content  analysis  and  policy-scaling 
techniques.  The  content  analysis  test  was  given  to  two  groups 
of  graduate  students,  one  an  ocean  policy  seminar,  the  other  a 
class  on  research  methods.  ^^Both  groups  had  the  equivalent  of 
one  class  period  of  introduction  to  the  methodology  of  the  LOS 
project.  That  is,  test  coders  were  minimally  "tutored."  The 
results  of  the  tests  can  thus  be  considered  a worst  case  approach 
to  the  reliability  of  the  project's  coding  techniques.  The  best 
case  is  represented  by  within-pro ject  coder  reliability,  which 
will  also  be  reported  for  comparative  purposes. 

The  policy-scaling  tests  were  given  to  a different  type 
of  group--sgbstantive  experts  on  Law  of  the  Sea  in  government 
and  academia.  The  intention  of  the  test  was  to  both  survey  the 
range  and  consistency  of  perceptions  of  LOS  issues  within  the 
ocean  policy  community,  and  to  test  the  sensitivity  of  the  LOS 
project's  forecasting  models  to  differential  perceptions  of  key 
issues . 

The  paper  will  also  compare  the  project's  final  forecasts 
(made  in  the  spring  of  1975)  with  the  Single  Negotiating  Text 
(SNT)  Issued  by  the  Geneva  session  of  the  LOS  Conference,  as 
a measure  of  the  validity  of  forecasts  that  were  made  using  the 
project's  models. 
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Before  proceeding  to  the  results  of  the  reliability 
tests,  a brief  explication  of  the  Project's  methodology  is  in 
order . 2 

LOS  PROJECT  METHODOLOGY 

The  project  collected  two  types  of  data  on  states  parti- 
cipating in  the  LOS  negotiations:  national  attribute  data  drawn 

from  standard  reference  works^  and  data  concerning  states'  posi- 
tions on  the  issues  under  negotiation,  drawn  from  official  UN 
verbatim  and  summary  transcripts  of  debates  in  the  UN  Seabed 
Committee  and  the  LOS  Conference,  and  supplemented  by  diplomatic 
reporting.  The  position  data  was  collected  by  means  of  thematic 
content  analysis. 

Generally  speaking,  content  analysis  is  "any  technique  for 
making  inferences  by  systematically  and  objectively  identifying 
specified  characteristics  of  messages."^  The  particular  approach 
adopted  by  the  LOS  Project  involved  the  extraction  of  themes — 
ideas,  complete  thoughts — from  the  basic  data  sources.  A theme 
was  defined  as  a statement  of  policy  on  a single  negotiating 
issue  by  an  official  country  representative,  or,  in  the  case  of 
diplomatic  reporting,  a policy  statement  attributed  to  a country 
or  its  representative  by  the  author  of  the  report. 


^This  description  of  the  methodology  draws  on  and  summar- 
izes a more  complete  discussion  found  in  Center  for  Naval  Analyses 
Research  Contribution  291,  "Forecasting  Outcomes  of  Multilateral 
Negotiations:  Methodology,  Volume  I:  Techniques  and  Models,"  by 

Robert 'L.'  Priedheim,  Karen  W.  Goudreau,  William  J.  Durch,  Unclas- 
sified, forthcoming. 

^A  list  of  sources  may  be  .found  in  the  appendix. 

^Ole  R.  Holsti,  "Content  Analysis,"  in  Gardner.  Lindsey 
and  Elliot  Aronson,  eds..  The  Handbook  of  Social  Psychology 
Volmne  Two;  Research  Methods,  2nd  edition,  (Reading,  Mass.: 
Addison-Wesley  Publishing  Co.,  1968),  p.  601. 
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The  system  used  an  open-ended  list  of  themes,  and 
separated  theme  identif ication  from  theme  categorization.  An 
open-ended  list  of  themes  allowed  coders  to  create  new  themes 
when  they  felt  that  ideas  encountered  in  the  source  materials 
were  not  adequately  covered  by  themes  already  in  use.  The  open- 
ended  list  facilitated  the  analysis  of  an  evolving  negotiation, 
in  which  the  negotiators  themselv^  were  breaking  new  conceptual 
ground  cver^’  year,  especially  in  the  late  1960s  and  early  1970s. 

For  exa  'oncepts  like  the  "common  heritage  of  mankind,"  the 

"inte  .ai  seabed  area,"  and  the  "exclusive  economic  zone" 

were  unknown  ten  years  ago.  Adding  new  themes  as  new  ideas  were 
encountered  allowed  the  scope  of  the  data  base  to  keep  pace  with 
the  scope  of  negotiations. 

Separating  theme  identification  from  categorization  reduced 
the  number  of  judgments  required  of  coders,  who  simply  scanned 
source  documents  for  codable  remarks  (policy  statements) , and 
determined  the  specific  theme  or  themes  expressed  in  the  remarks. 
Every  theme  was  assigned  its  own  code  number.  When  coders 
encountered  remarks  expressing  a particular  theme  (for  example, 
support  for  a 12-mile  limit  to  the  territorial  sea,  theme  number 
934),  they  would  mark  that  number  in  the  margin  of  the  source 
document,  opposite  the  remark.  When  the  document  was  completely 

5 

coded  and  quality-checked,  theme  codes  were  recorded  on  card 

^A  ten  percent  or  greater  sample  of  coding,  stratified  by 
issue,  was  checked  by  a senior  coder,  or  a project  analyst,  before 
a round  of  coding  was  added  to  the  data  base.  Errors  found  were 
corrected,  and  large  error  rates  triggered  complete  recoding. 
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layout  shift  ^ :.j  with  the  code  number  of  the  country  speaking 


anu  cttrtain  ■ ' reference  codes,  keypunched,  and  stored  on  com- 
puter ta^'.  " t ater  analysis.  Using  this  coding  system,  the  pro- 
ject assert)  • d i data  bank  of  45,000  observations  on  1,500  themes, 
covering  a t year  period  from  late  1967  through  spring  1975. 


As  r : nt  of  themes  to  categories  was  done  in  a separate 
process,  riiar  ut  creating  policy  scales  or  " issue-variables"  — 
so  called  '>or:i,-se  they  constitute  the  basic  conceptual  framework 
along  which  national  positions  on  the  issues  vary  in  their  values. 6 
To  cro  ;te  an  issue-variable,  all  themes  relating  to  a 
particular  policy  question  under  negotiation  are  aggregated,  and 
then  ordi  i'e  ’ . nto  a policy  continuum.  For  example,  themes  relat- 
ing to  ter  itorial  sea  delimitation  might  be  ordered  according  to 
breadth  ‘ proposed."^ 

Support  a "narrow"  territorial  sea 
Support  a 3-mile  territorial  sea 
: upport  a 12-mile  territorial  sea 
Support  for  12  miles  tied  to  a 
200-mile  economic  zone 
Support  a 50-mile  territorial  sea 
iach  state  has  a right  to  delimit  its  own 
sovereignty  zone  unilaterally 
ihjpport  a 200-mile  territorial  sea 

The  Sits;  of  ordered  themes  are  then  policy  spaced.®  Scale 

values  are  unsigned  each  theme  in  a set  according  to  the  degree 

of  substantive  difference  seen  between  adjacent  themes  in  the 

ordered  set.  The  smallest  substantive  difference  between  two 


definition  suggested  by  J.  Christian  Kessler,  currently 
with  Lulejian  and  Associates,  Arlington,  Virginia. 

^Thj.s  example  uses  only  one-sixth  of  the  themes  contained 
in  the  delimitation  variable  used  by  the  project. 

®Thn  syistem  used  by  the  LOS  project  generally  corresponds 
to  what  Torgerson  calls  the  subjective  estimates/arrangements 
method.  See  Warren  S.  Torgerson,  Theory  and  Methods  of  Scaling, 
(New  York:  Wiley,  1958),  Chapters  1 ana  4 . 
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themes  X and  Y that  would  have  non-trivial  consequences  for 
policy  (it  would  in  fact  matter  if  Y were  written  into  a treaty 
instead  of  X)  is  represented  by  a difference  of  one  whole  number 
in  the  scale  values  assigned  X and  Y.  Functionally  equivalent 
proposal.,  would  be  assigned  identical  scale  values,  while  those 
seen  to  differ  markedly  would  be  separated  by  two  or  more  numbers. 
The  above  set  of  ordered  themes  might  be  policy  spaced  as  follows: 


Scale 

Theme  Value 


Support  a "narrow"  territorial  sea  1 
Support  a 3-mile  territorial  sea  1 
Support  a 12-mile  territorial  sea  3 
Support  for  12  miles  tied  to  a 

2^^0-mile  economic  zone  ^ 
Support  a 50-raile  territorial  sea  9 
Each  state  has  a right  to  delimit  its  own 

sovereignty  zone  unilaterally  17 
Support  a 200-mile  territorial  sea  20 


The  first  two  themes  differ  insignificantly.  Twelve  miles  is 
the  currently  designated  replacement  for  the  three-mile  limit. 

Had  limits  of  six  or  eight  miles  been  included  in  this  set,  they 
would  have  been  scaled  at  two.  The  theme  expressing  condi- 
tional support  for  12  miles  was  placed  at  4 because  countries 
using  it  were  offering  to  trade  a broad  territorial  sea  (up 
to  200  miles)  for  a narrow  sea  plus  an  economic  zone.  Themes 
expressing  support  for  much  broader  limits  are  placed  corres- 
pondingly further  down  the  scale. 

Problems  in  scaling  arise  when  there  are  themes  in  the 
set  whose  substantive  impact  is  difficult  to  assess.  In  the  above 
set,  unilateral  delimitation  is  such  a theme.  It  was  placed  far 
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down  the  scale  both  because  it  represents  a policy  that  would 
allow  every  country  to  declare  a very  broad  territorial  sea,  and 
because  the  concept  was  used  as  a kind  of  codeword  by  countries 
interested  in  establishing  such  broad  zones.  Thus  the  placing 
of  such  ambiguous  themes  is  in  part  dependent  upon  some  knowledge 
of  how  they  were  used  in  the  actual  negotiations.  There  is  some 
danger  inherent  in  this  approach,  of  course,  in  that  one  might 
begin  to  scale  themes  less  on  the  basis  of  what  the  *-hemes  say 
than  on  the  basis  of  who  said  them;  and  that  could  result  in 
distorted  calculations  of  countries'  positions  on  the  issues. 

We  will  return  to  the  question  of  ambiguous  themes  in  evaluating 
the  scaling  tests. 

Due  to  that  potential  for  bias,  policy  spacing  on  the  LOS 
project  was  a collegial  process,  and  the  final  policy  scale  for 
each  issue  was  the  product  of  consensus  among  judges.  However, 
because  a process  of  socialization  within  the  project  may  have 
created,  in  effect,  a project  bias  on  some  issues,  we  endeavored 
to  make  the  scaling  process  as  "public"  as  possible.  That  is, 
whenever  possible,  decision-makers  were  asked  to  review  the  issue 
variables,  and  indicate  where  their  perceptions  of  the  issues 
departed  from  our  own.  Where  they  differed  in  important  respects 
comparative  analyses  were  run,  the  results  of  which  showed  the 
consequences,  if  any,  of  differing  perceptions.  In  addition, 
the  issue-variables  underlying  each  analysis  were  reported  with 
the  analysis,  for  the  inspection  of  users. 
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A country's  position  on  an  issue  is  defined  as  the  mean 
of  its  remarks  on  that  issue,  as  measured  on  the  relevant  policy 
scale.  Thus  if  Australia  supported  a 12-mile  territorial  sea 
once,  and  gave  conditional  support  another  time,  its  "national 
score"  for  territorial  sea  delimitation  would  be  (3+4)/2  = 3.5. 

As  a mean,  this  figure  can  be  interpreted  as  the  expected  value 
of  Australia's  position  on  the  territorial  sea,  based  on  its 
remarks  for  the  record. 

Since  there  are  always  some  countries  who  do  not  address 
some  of  the  issues  during  a session  of  the  Conference  (or  during 
whatever  time  period  is  being  used  for  analysis) , there  are  gaps 
in  the  sets  of  national  scores.  Since  one  of  the  goals  of  the 
project  was  forecasting  the  votes  of  all  eligible  participants 
in  the  negotiations,  those  missing  aata  were  fillea  in  oy 
means  of  multiple  regression. 

A separate  regression  is  run  for  each  issue-variable  in  the 
analysis.  Each  country  with  a national  score  constitutes  a 
case  in  the  regression.  The  national  score  is  the  dependent 
variable,  and  dichotomized  national  attribute/interest  variables 
are  the  predictors. 

Countries  receive  a score  of  one  (1)  for  those  attribute/ 
interest  variables  whose  inclusion  criteria  they  meet,  and  a 
score  of  zero  (0)  on  the  remainder.  Sample  variables  (of  a total 
of  33  available  for  use  by  the  regression  routine)  are  given  in 

9 

table  one,  below. 

9 

A complete  list  is  gi”en  in  the  appendix. 
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TABLE  1 


Variable 
Major  oil  producer 

Major  mineral  producer 


Major  merchant  fleet 


Group  of  77 


Broad  shelf 


Africa,  Asia,  Latin 
America,  Eastern  Europe, 
Western  Europe  and  others 


Criterion  for  inclusion 

Oil  production  of  at  least  100,000 
barrels  per  day 

Accounts  for  at  least  2%  of  world 
production  of  copper,  nickel,  manga- 
nese, or  cobalt.  (Minerals  found 
in  seabed  nodules.) 

At  least  5 million  deadweight  tons 
of  shipping  fly  the  country's  flag. 

Member  of  the  UN's  "Group  of  77" 
developing  nations. 

Countries  with  wider  than  average 
(30  nautical  miles)  continental 
shelves. 

UN  political  groups.  Each  country 
belongs  to  one  and  only  one. 


The  feq teas ion  is  forced  through  the  origin,  distributing  the 


constant  term  to  a set  of  variables  that  represent  the  United 
Nations  regional  caucusing  groups  (Africa,  Asia,  Latin  America, 
Eastern  Europe,  and  a residual  category,  Western  Europe  and  Others). 
Membership  in  these  groups  is  mutually  exclusive  and  exhaustive. 

The  regression  coefficient  for  region  thus  constitutes  each 
country's  "base"  estimate.  Its  other  interests  and  attributes 
act  to  effect  marginal  changes  in  that  estimate.  Up  to  25  vari- 
ables are  allowed  in  the  equation.  Those  entering  in  the  latter 
steps  of  the  regression  make  minimal  contributions  to  variance 
explained.  They  are  included  to  allow  the  model  to  discriminate 
as  much  as  possible  among  countries  with  similar,  but  not 
identical,  interests. 

The  non-standardized  regression  coefficients  generated  by 
each  regression  are  used  to  calculate  estimated  scores  for  every 
country — those  that  served  as  cases  in  the  regression  and  those 
that  did  not  (countries  without  national  scores) . The  estimate 
for  a given  country  is  simply  the  sum  of  the  coefficients  of 
those  variables  in  the  regression  for  which  it  has  a score  of 
one,  as  shown  in  table  two.'*'® 

^^The  table  contains  only  those  variables  from  the  regres- 


sion to  which  at  least  one  of  the  sample  countries  belong.  Thus 
two  regions,  Asia  and  Eastern  Europe,  are  not  represented 
although  they  were  included  in  the  regression. 


TABLE  2 ! CALCULATION  OF  REGRESSION  ESTIMATES 
(ISSUE-VARIABLE:  TERRITORIAL  SEA) 


Regression  Coeffs: 

1 .7f> 

-3.2 

-2.6 

-7.C* 

-1.5 

3.6 

2.4 

12.6 

15.2 

6.7 

-L6 

-2.1 

Scores  on  Indep. 
Vdr  lables : 

Austra I la 

1 

1 

1 

1 

0 

1 

1 

0 

0 

0 

1 

1 

0 

0 

Botswana 

0 

0 

0 

1 

0 

0 

0 

1 

0 

0 

0 

0 

0 

Chad 

0 

0 

0 

1 

0 

0 

0 

1 

0 

0 

0 

0 

0 

Ecuador 

1 

0 

0 

0 

0 

0 

1 

0 

1 

0 

0 

1 

I 

National 


Scores 

Est imates 

Residuals 

Austral la 

3.5 

1.76*(-3.2)+(-2 

.6)+(-l. 

,5)+3.6+6.7*(-1.6)+2. 

.3  - 5.46 

-1.96 

Botswana 

3.0 

(-;.9)+12.6 

- 4.7 

-1.7 

Chad 

— 

(-7.9)*12.6 

- 4.7 

Ecuador 

19.0 

1.76  + 2.4<-15.2  + (- 

-2.1) +2. 

.3 

- 19.56 

-0.56 

This  use  of  regression  to  fill  in  missinq  data  assumes 
that  states  with  like  attributes  and  interests  will  take  similar 
negotiating  positions.  Thus  the  procedure  generates  identical 
estimates  for  Botswana  and  Chad,  because  their  interests  in  the 
model  are  identical — both  are  landlocked  African  countries. 

Once  regression  estimates  have  been  generated,  final  indi 
cators  of  countries'  positions  on  the  issues  are  calculated. 

For  countries  lacking  national  scores,  these  final  indicators, 
or  "preferred  positions,"  equal  the  regression  estimates.  For 
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countries  with  national  scores,  preferred  positions  are 
calculated  as  a weighted  av'erage  of  score  and  estimate: 

A 

Y N + Y 

Preferred  Position  = ij  ij  ij  , where 

N + 1 

ij 


Y. . = National  score  for  country  j on  issue- 
variable  i. 

A 

Y^.  = Regression  estimate  for  country  j on  issue- 
^ variable  i. 


N^.  = Number  of  remarks  by  country  j on  issue- 
^ variable  i. 

In  calculating  preferred  positions,  the  national  score  is 
weighted  by  the  number  of  remarks  upon  which  it  was  based,  and 
the  estimate  is  given  a weight  of  one.  This  has  the  effect  of 
both  Introducing  a " group" inf  1 uence  into  a country's  position, 
and  correcting  for  stray  remarks.  The  potential  effect  of  averag- 
ing in  the  estimate  is  inversely  proportional  to  the  number  of 
remarks  weighting  the  national  score  ("potential"  because  if 
score  and  estimate  are  close,  the  effect  will  be  nil,  whatever 
the  number  of  remarks;  we  are  essentially  speaking  of  the  effect 
on  states  with  large  residuals).  Operationally,  averaging  in  the 
estimate  serves  to  move  a country's  preferred  position  in  the 
direction  of  the  preferred  positions  of  those  countries  with  whom 
it  shares  interests.  The  estimate  also  serves  to  cushion  the 
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effect  of  idiosyncratic  remarks  which  may  not  reliably  reflect 
a country's  negotiating  position,  a problem  when  a national 
score  is  based  on  a single  remark.  This  assumes  that,  given  a 
large  residual  in  such  a case,  the  regression  estimate  reflects 
the  country's  "true"  interests  better  than  its  own  statement  of 
policy;  or  at  least  that  the  final  preferred  position  does  so — 
admittedly  a large  assumption,  but  one  we  have  found  to  give 
reasonable  results. 

The  media;',  of  the  distribution  of  preferred  positions  on 
a given  issue-variable  is  generally  used  as  the  estimate  of  most 
likely  outcome  on  that  issue.  The  median  is  used  in  preference 
to  the  mean  because  its  calculation  gives  equal  weight 
to  scores,  analogous  to  one-state-one-vote,  whereas  the  mean  would 
assign  proportionately  greater  impact  on  outcome  to  countries  with 
positions  toward  the  tails  of  the  distribution.  It  also  consis- 
tently minimizes  the  aggregate  movement  of  countries  away  from  their 
preferred  positions.  Finally,  voting  theory  suggests  that  the 
median  is  that  outcome  against  which  no  other  outcome  can  win, 
under  majority  voting  — provided  only  that  all  states  vote  their 
indicated  preferences,  and  no  deals  are  made.^^ 


11 

See  Duncan  Black,  The  Theory  of  Committees  and  Elections 
(Cambridge;  Cambridge  University  Press,  1958) . 
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Thero  dre  a number  of  idditionai  forecasting  models 

incorporated  in  the  methodology  of  the  LOS  pro j ect--among  them 

utility  models  for  sinqle  issues  and  packages  of  issues;  and  a 

program  to  maximize  the  number  of  supporters  for  any  given 

package  of  proposals--they  go  beyond  the  scope  of  the  tests 

12 

reported  in  this  paper.  We  turn  now  to  those  tests. 

CODING  itdL  [AiJiLl'iY 

The  coding  test  consisted  of  12  composite  "speeches" 

drawn  from  United  Nations  summary  records  of  the  Caracas  session 

of  the  LOS  Conference.  Speeches  were  compiled  so  that  each  one 

addressed  three  issues  and  (if  possible)  no  others:  territorial 

sea  delimitation,  the  question  of  powers  to  be  given  the  proposed 

International  Seabed  Authority,  and  the  question  of  jurisdiction 

over  scientific  research  in  coastal  waters.  Countries  were 

chosen  so  as  to  represent  all  five  regional  caucusing  groups, 

and  were  stratified  according  to  development  status  and  whether 

13 

or  not  they  were  "geographically  disadvantaged."  Countries 
included  in  the  test  are  listed  in  table  3,  below. 

^^They  are,  however,  fully  discussed  in  CNA  Research  Contribu- 
tion 291, chapter  three. 

^ ^The  term  "disadvantaged"  applies  to  countries  that  are  either 
landlocked,  or  shelflocked — unable  to  claim  an  offshore  area 
extending  beyond  the  200-meter  isobath.  "Advantaged"  states 
have  open  coastlines  and  can  make  more  extensive  claims. 
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TABLE  3 


COUNTRIES  INCLUDED  IN  CODING  TEST 


Developing 


Developed 


Each  cell  in  table  three  constitutes  a country  type  for 
purposes  of  this  test. 

The  introduction  to  the  test  briefly  explained  the  coding 
methodology,  and  then  asked  readers  to  code  a sample  speech  as 
an  exercise  in  learning  by  doin’g.  Coders  were  given  a codebook 
listing  themes  and  theme  numbers  for  each  of  the  test  issues, 
which  was  to  be  used  in  coding  the  test.  However,  coders  were 
instructed  to  add  themes  of  their  own  to  the  codebook  if  they 
felt  that  themes  in  the  codebook  did  not  adequately  reflect 
remarks  found  in  the  speeches. 

The  test  was  given  to  three  separate  groups:  a graduate 

class  in  research  methods  (38  tests  returned),  a graduate  seminar 
in  ocean  policy  (eight  tests  returned) , and  four  members  of 
the  LOS  project.  The  last  served  as  the  control  group  for  evalu- 
ating the  students'  performance.  Both  groups  of  students  had 
one  class  meeting  devoted  to  an  explanation  of  the  LOS  methodology. 


Geographically  Geographically 

disadvantaged  advantaged 


Bolivia 

Ecuador 

Iraq 

India 

Trinidad 

Nigeria 

Austria 

Canada 

Finland 

Japan 

Germany,  GDR 

United  States 
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Thus  compared  to  the  control  group,  they  were  relatively  untutored 
in  the  methodology.  The  methods  class  was  not  socialized  to 
the  norms  and  perceptions  of  the  project.  The  ocean  policy 
seminar,  however,  was  taught  by  the  project  director,  Robert 
Friedheim,  and  must  be  considered  partially  socialized. 

Coders  were  given  a questionnaire  to  complete  after  they 
had  finished  the  test.  They  were  asked  to  give  the  amount  of 
time  taken  to  complete  the  test,  to  indicate  their  level  of 
knowledge  of  law  of  the  sea  issues,  and  to  rate  the  coding 
instructions.  Time  (two  hours  on  average)  and  self-rated  level 
of  knowledge  (medium)  did  not  vary  sufficiently  to  be  useful  as 
control  variables.  However,  the  six  coders  who  rated  the 
instructions  "inadequate"  were  dropped  from  the  test.  The  results 
given  below  are  thus  controlled  for  sheer  confusion.  Coders 
evaluated  at  least  believed  they  knew  what  they  were  doing. 

Denying  test  results  for  laoth  experienced  and  inexperienced 
coueis  should  enable  us  to  establish  the  upper  and  lower  bounds 
of  reliability  for  the  project's  content  analysis  techniques. 
Results  for  the  control  group  will  thus  be  reported  along  with 
those  for  the  test  groups. 

The  test  speeches  were  brief,  averaging  22  lines  each 
(the  shortest  was  13  lines,  the  longest  27).  The  order  of  issues 
within  speeches  varied,  and  the  order  of  countries  was  such 
that  country  types  were  evenly  distributed  throughout  the  test. 


-16- 


Coders  were  asked  to  first  bracket  remarks  they  considered 
codable.  This  procedure  allowed  us  to  transform  the  completed 
tests  into  a three  dimensional  data  matrix,  countries  by  coders 
by  theme-cells-within-countries . Each  theme-cell  was  assigned 
a score  of  zero  or  one,  depending  on  whether  ot  not  it  represented 
a codable  remark  in  the  original  text.  Scores  were  derived 
from  the  tests  taken  by  the  control  group  of  LOS  project  coders. 
Remarks  which  were  bracketed  and  coded  by  two  or  more  LOS  coders 
were  defined  as  "codable"  and  those  theme-cells  received  scores 
of  one.  All  other  cells,  including  those  rated  codable  by  just 
one  LOS  coder,  were  assigned  a score  of  zero. 

The  tests  were  evaluated  to  determine  the  proportion  of 
codable  remarks  missed  by  the  student  coders  (the  rate  of  omis- 
sion or  undercoding)  as  well  as  the  proportion  of  remarks  coded 
that  the  control  group  did  not  consider  codable  (the  rate  of 
commission  or  overcoding) . National  scores  were  then  calculated 
for  each  country,  by  issue  and  coder.  Differences  were  taken 
between  these  and  the  base  national  scores  for  each  country /issue 
derived  from  the  control  group's  tests.  (The  base  national 

score  for  India  on  deep  seabeds , for  example,  was  the  grand  mean 
of  all  deep  seeibed  themes  assigned  India  by  the  LOS  coders.) 

While  the  calculations  of  differences  were  straightforward,  the 
calculation  of  under-  and  overcoding  rates,  and  the  reasoning 
behind  it,  requires  greater  explanation. 
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The  overall  results  of  the  codinq  test  can  be  simply  dis- 
played in  a 2 X 2 contingency  table  (table  4) . Cell  one  repre- 
sents joint  agreement — numbers  of  codable  remarks  caught  by 
test  coders.  Cell  two  represents  undercoding--codable  remarks 
missed.  Cell  three  represents  overcod ing--extraneous  themes. 
Cell  four  is  agreed  non-acts,  and  as  such  cannot  be  readily 
defined . 

TABLE  4 

CODING  TEST  CONTINGENCY  TABLE 

Test  Coders  Results 


Did 

code 

Did  not 
code 

Control 

fjrnnp 

Should  code 

1/ 

1859 

^ 474 

results 

Should  not 
code 

2/ 

334 

i/ 

To  evaluate  these  results,  one  might  simply  calculate 
the  proportion  of  remarks  in  each  cell,  that  is,  70%  of  all 
remarks  in  the  table  were  correctly  recognized;  18%  were  missed; 
and  13%  were  extraneous.  However,  that  approach  uses  an 
inflated  denominator  to  calculate  error  proportions.  The  under- 
coding score  in  particular  is  deflated  to  the  extant  that  there 
are  overcoding  errors.  (The  reverse  is  not  the  case.  Fewer 
undercodings  mean  greater  joint  agreement,  leaving  the  denominator 
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unchanged.)  A better  approach  to  undercoding  would  be  to  calcu- 
late the  proportion  of  valid  themes  missed  by  test  coders,  that 
is, 


underccding  rate  = 5gn-°-r-‘  -c-5-lTT  ‘ • 

By  the  same  token,  overcoding  would  be  better  represented  as  the 
proportion  of  overcodes  among  all  test  coding  acts.  Since  cell  2 
constitutes  test  coder  non-acts,  the  equation  would  be. 

Overcoding  rate  = ceTl'''l^V  cell  3 =-15. 

Using  the  frequency  distribution  in  table  4,  above,  we  arrive 
at  an  undercoding  rate  of  20%  (overall,  one  out  of  five  valid 
themes  was  missed  by  test  coders) ; and  an  overcoding  rate  of 
15%  (overall,  about  one  out  of  every  seven  remarks  assigned 
themes  by  test  coders  was  not,  in  fact,  codable,  according  to 
the  control  group) . 

The  chief  advantage  of  this  approach  is  that  both  rates 
can  be  interpreted  as  simple  probability  statements.  The  chief 
disadvange  for  those  who  prefer  single  summary  statistics, 
is  that  it  disaggregates  error  into  two  measures  that 
cannot  be  readily  combined  (due  to  differing  denominators.) 

The  combined  error  rate,  in  other  words,  is  not  35%.  The 
measures  constitute  two  distinct  cuts  at  the  codina  results 
and  should  be  considered  separately.  Indeed,  more  informa- 
tive than  their  combined  values  are  their  relative  dis- 
tributions across  issues  and  country  types  (which  will  be 
noted  below)  and  what  those  distributions  say  about 
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systematic  error  in  the  coding.  Compared  to  the  first  approach 
of  measarinq  the  proportional  distribution  of  coding  results 
in  tabl.  4,  the  dual  approach  is  relatively  conservative,  con- 
stituting a worst  case  approach  to  recognition  error. 

As  noted  above,  over  all  issues  and  countries,  test 
coders  averaged  20%  undercoding  and  15%  overcoding.  Comparable 

figures  for  the  control  group  (LOS  project  coders)  were  15%  and 

14 

7%,  respectively.  Coders  using  the  LOS  project's  content  analy- 
sis methods  might  be  expected,  then,  to  catch  80-85%  of  all  codable 
remarks  in  a given  set  of  documents,  and  to  actually  assign  themes 
to  codable,  as  opposed  to  extraneous  remarks  85-93%  of  the  time 
(see  table  5) . Note  in  table  5 that  errors  of  omission  are 
greater  than  commission  for  both  the  test  and  control  groups. 
Indeed,  the  difference  in  rates  is  more  marked  in  the  control 
group:  while  still  small,  omissions  are  double  commissions. 

This  is  a persistent  problem  in  thematic  analysis,  because  the 
units  of  analysis  (themes)  have  no  natural  boundaries.  Thus 
coders  tend  to  code  too  few,  rather  than  too  many,  remarks. 

There  is  a good  deal  of  variation  by  issue  in  both  types 
of  error,  and  a certain  amount  of  variation  by  country  type. 

Test  coders  had  least  difficulty  with  territorial  sea,  more 
difficulty  with  science,  and  most  witli  the  deep  seabeds.  The 

^^LOS  coders'  error  rates  were  calculated  using  the  same  evalua- 
tion matrix  used  for  the  student  coders.  Thus  if  two  of  the 
four  project  coders  rated  a given  remark  as  codable,  and  two 
did  not,  two  omission  errors  resulted.  Remarks  coded  by  just  one 
project  coder  constituted  overcoding. 
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TABLE  5 


THEME  RECOGNITION  ERROR  RATES 
A.  UNDERCODING  [.20  (.15)]* 


Issues:  Terr.  Deep 

sea  Science  seabeds 


Country 

types 

Overall 

scores 

.12 

( .04) 

.19 

(.22) 

.27 

(.15) 

Disad . 

.25 

(.20) 

.09 

( .08) 

.18 

(.19) 

. 36 

(.26) 

Developing 

Advan . 

.21 

(.11) 

. 26 

(.05) 

.06 

(.10) 

. 31 

(.17) 

Disad . 

.19 

(.13) 

.03 

(0) 

. 31 

( .25) 

.07 

(0) 

Developed 

Advan . 

.16 

(.16) 

. 0 . 

(0)  . 

.18 

( .29)  . 

.22 

( .10) 

B.  OVERCODING  [.15  (.07)]* 

Issues:  Terr.  Deep 

sea  Science  seabeds 


Country 

types 

Overall 

scores 

.03 

(.04) 

.17 

(.07) 

.21 

(.09) 

Disad . 

.13 

(.08) 

0 

(0) 

.18 

(.14) 

.16 

(.05) 

Developing 

Advan . 

.12 

(.07) 

0 

(.05) 

.08 

(0) 

.24 

(.05) 

Disad . 

D 

. 11 

(.08) 

. 14 

(0) 

.26 

(.08) 

Developed 

■1 

Advan. 

■ 

0 

(0) 

.27 

(.11) 

.20 

(.18) 

*Numbers  in  parentheses  are  scores  for  the  control  group. 
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increasinq  error  rate  parallels  the  increasing  technical  complex- 
ity of  tht;  issues.  The  first  is  a simple  boundary  question, 
and  policy  statements  relating  to  it  are  brief  and  straight- 
forward. .Science  involves  the  degree  of  coastal  state  juris- 
diction over  research,  while  the  third  question  combines  the 
issues  of  access  to  seabed  resources  and  powers  to  be  accorded 
the  proposed  international  seabed  minerals  agency.  While  control 
group  undercoding  errors  are  more  frequent  for  the  latter  two 
issues  than  for  territorial  sea,  overcoding  remains  low  for 
all  three.  Additional  experience,  in  other  words,  seems  to 
effectively  teach  coders  what  not  to  code.  It  also  reduces 
errors  of  omission,  but  not  to  the  same  degree. 

Test  results  also  vary  by  country  type.  More  accurately, 
they  vary  according  to  whether  a country  is  developed  or  develop- 
ing. Test  coders  tended  to  miss  policy  statements  by  LDCs  more 
frequently  than  they  missed  statements  by  developed  countries. 

They  also  coded  fewer  extraneous  remarks  for  LDCs.  Overall, 
then,  test  coders  seemed  to  devote  more  attention  to  speeches 
by  developed  countries,  coding  them  more  intensively.  This 

differential  coding  suggests  a certain  amount  of  cultural  bias 

15 

in  test  coders,  relative  to  the  control  group. 

1‘^It  should  be  noted  that  neither  of  the  patterns  discussed 
above  (variation  in  error  rate  by  issue  and  country  type) 
are  as  evident  when  issues  and  countries  are  cross- tabulated . 
Clearly,  the  relatively  small  number  of  countries  in  the  test, 
allowing  only  three  countries  per  "type"  may  permit  sufficiently 
strong  interaction  effects — due  to  idiosyncratic  structure  or 
style  of  individual  speeches,  for  example--to  masquerade  as 
main  effects.  For  the  time  being,  this  problem  can  only  be 
acknowledged,  and  the  suggestion  made  that,  if  country  effects 
are  uncertain,  issue  effects  appear  to  be  genuine,  confirmed 
by  analysis  of  differences  in  national  scores  below. 
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Table  6 compares  recognition  results  from  the  ocean 
policy  seminar  with  those  from  the  research  methods  class. 
The  sample  size  from  the  policy  seminar  is  small  (7  coders) 


M 


► 


SO  conclusions  drawn  from  this  comparison  are  tentative.  The 
partially  socialized  coders  from  the  seminar  seemed  to  have 
a better  idea  of  what  not  to  code,  but  had  a slightly  higher 
omissions  rate.  Overall they  tended  to  be  less  aggressive 
coders  than  the  methods  class.  Both  groups,  however,  exhibit 
the  same  tendency  to  undercode  remarks  by  LDCs. 


TABLE  6:  RECOGNITION  ERROR  RATES, 
BY  TEST  GROUP 


Undercoding 


Overcoding 


Terr . 
Sea 

Science 

Seabeds 

Terr . 
Sea 

Science 
1 

Seabeds 

Pol icy 

.14 

.17 

.32 

.01 

HQ 

.17 

Methods 

.11 

.20 

.26 

.03 

.17 

.22 

Undercod ing 


Overcod ing 


Developing 

Developed 

Developing 

Developed! 

Dis . 

Ad  V . 

Dis. 

Adv. 

mm 

Adv . 

Adv. 

Pol icy 

.32 

.23 

.17 

D 

.14 

.08 

.13 

.14 

Methods 

.23 

.21 

.20 

.16 

.11 

.11 

.19 

.20 

Differences  between  test  and  l)ase  scores  were  assessed  two  ways 
by  .ibsolute  mean  difference,  and  by  direction  of  difference 
(table  7) . A difference  of  4 scale  values  or  more  between  a 
coder's  national  score  and  the  appropriate  "base"  was  considered 
a serious  error.  Such  a difference  would  result  in  a substan- 
tively different  interpretation  of  a country's  position  on 
any  one  of  the  three  test  issue-variables . 

TABLE  7 

DlrrERRNCES  IN  NATIONAL  SCORES, 

TEST  CCXIERS  AND  LOS  CODERS 


A.  Absolute  Meun  Differences 


Issues ; 

Tejfr.  Sea 

Science 

Scabeds 

Mean  Difference 

.50  (.06) 

1.5  (.80) 

2.3  (.00) 

Percent  Serious 

2%  { 0 ) 

10%  { 0 ) 

o 

CD 

Di  f ference 

Country  Types: 

LUC 
Di  s . 

LDC 

Adv, 

DC 
D IS  . 

DC 

Adv. 

Kean  Difference 

1.0  (.5) 

1.6(.6) 

1.3  (.5) 

1 .4  (.6  ) 

Percent  Serious 
Dif  ference 

10% 

9% 

11% 

9% 

h ♦ 

B . Distribution  of  Difference r 


All  Differences 
(1  unit  or  greater) 

SeriDU.s  Differences 
(4  units  or  greater) 

T.S  . 

Sex . 

SB 

T.S  . 

Sci. 

SB 

AH  Test  Coders 

2 0% 

59% 

58% 

100% 

60% 

66% 

Policy  Seminar 

2 6% 

66% 

66% 

100% 

0 

84% 

Methods  Class 

76% 

60% 

57% 

100% 

66% 

6 3% 

* 

LOS  project  coders’  differences,  also  calculeted  from  the 
^^bJise  tiational  scores,  are  given  m parentheses. 

Cell  entries  are  percent  of  cases  in  which  tost  coders' 
national  scores  were  smaller  --  more  t'jward  t lie  developed 
state  end  of  the  policy  scale  --  than  the  base  scores. 
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As  with  recognition  errors,  differences  in  scores  vary 
by  issue,  in  the  same  issue  order.  The  mean  difference  for 
territorial  sea  was  less  than  one  unit,  an  uninterpretable 
difference.  Serious  errors  ran  2%.  Differences  in  science 
scores  averaged  1.5  units,  with  10%  serious  error.  Seabeds 
scores  were  off  2.3  units,  and  serious  errors  ran  18%.  Still, 
even  in  the  last  case,  four  out  of  five  coders  scores  fell 
within  four  units  of  the  base  scores,  an  acceptable  outcome 
. especially  considering  that  test  coders  did  not  know  the 
scale  values  of  any  of  the  themes  they  assigned. 

However,  test  coders  did  tend  to  assign  themes  that  were 
ranked  more  toward  the  developed  state  end  of  the  three  issue- 
variables  (smaller  scale  values)  relative  to  themes  assigned  by 
the  control  group.  That  is,  they  tended  to  interpret  remarks 
as  being  more  favorable  toward  narrower  territorial  seas, 
freedom  of  science,  and  limited  powers  for  the  seabed  agency, 
than  did  LOS  project  coders.  If  this  does  not  again  suggest 
cultural  bias  in  test  coders,  it  at  least  suggests  that 
experience  and  socialization  sensitizes  the  coder  to  third  world 
policies  and  reduces  the  tendency  to  reinterpret  unfamiliar 
statements  according  to  familiar  frames  of  reference. 

Table  7b  shows  that  interpretable  differences  in  national 
scores  (those  of  one  unit  or  greater),  with  the  exception  of 
territorial  sea,  where  10  of  12  base  scores  were  3.0,  tilted 
toward  the  developed  state  end  of  the  scale  by  about  60%  to  40%. 
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The  distribution  of  serious  differences  was  even  more  skewed, 
favoring  developed  state  positions  by  2:1.  The  distribution  of 
differences  was  about  the  same  for  both  test  groups. 

Country  type  seems  not  to  have  affected  this  process. 
Developed  and  developing  countries  alike  were  shifted  toward 
the  developed  state  end  of  the  policy  scales  by  test  coders. 

To  summarize  the  results  of  the  coding  test,  we  found 
that  minimally  trained  coders  can  use  the  content  analysis 
methodology  with  very  reliable  results.  The  chief  benefits  of 
increased  coder  socialization  were  found  in  reduction  of  over- 
coding and  cultural  bias. 

POLICY  SCALING  RF'.I,IABTT,ITY 

Next  to  the  data  itself,  the  policy  scales  (issue  variables) 
developed  to  measure  countries'  negotiating  positions  on  the 
issues  are  the  most  important  variables  affecting  the  project's 
analyses  of  negotiations  and  forecasts  of  negotiating  outcomes. 
Those  forecasts  are  valid  only  to  the  extent  that  the  underly- 
ing issue-variables  reflect  reasonably  well  the  way  negotiators 
perceive  the  issues  and  the  relationships  of  alternative  solu- 
tions to  one  another.  Whether  the  issue-variables  do  so  depends 
in  part  on  our  perceptions  of  the  issues,  and  in  part  on  the  method 
of  scaling  we  have  developed  to  reify  those  perceptions.  To 
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test  the  reasonableness  of  LOS  project  analysts'  perceptions 
of  the  issues  and  the  reliability  of  the  scaling  methods, 
we  consulted  law  of  the  sea  experts  outside  of  CNA  by  means  of 
a three-issue  scaling  test.  Seventeen  tests  were  sent  out  to 
experts  in  government  and  academe;  ten  were  returned  and  consti- 
tute the  sample  evaluated  here. 

The  three  issues  were  chosen  to  be  representative  of  the 

scope  of  negotiations  at  the  LOS  Conference.  Thus  we  included 

the  question  of  deep  seabed  mining,  a navigation  issue  (straits 

transit) , and  a coastal  state  jurisdiction  issue  (scientific 

\ 

research  in  the  coastal  zone) . Twelve  themes  were  selected, 
representing  the  full  range  of  proposals  tabled  during  the 
negotiations,  for  each  issue.  Themes  were  printed  on  cards. 

Thus  each  participant  received  a dec)c  of  36  cards  arranged  into 
three  12-card  issue-subsets.  Card  order  within  subsets  was 
randomized  across  participants  to  avoid  primacy-latency  effects 
on  ordering. 

Participants  were  given  a set  of  written  instructions 
that  explained  how  policy  scales  were  created  and  used  by  the 
project.  The  instructions  also  gave  step-by-step  proced  .res 
for  ordering  and  scaling.  All  participants  rated  the  instruc- 
tions at  least  "adequate."  Following  Torgerson's  suggestion. 
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tht  bc.tliny  criterion  and  the  policy  endpoints  for  each  issue 
were  specified  (table  8) , as  was  the  origin  (a  scale  value  of 
one).^^  Participants  were  given  the  option  of  rescaling  any 
of  the  test  issues  according  to  a different  criterion  if  they 
felt  the  one  given  in  the  test  was  inappropriate.  (None  did  so.) 
Identical  tests  were  given  to  three  LOS  project  analysts  for 
purposes  of  direct  comparison  with  outside  test  participants' 
results . 

The  tests  are  evaluated  here  in  several  ways,  proceeding 
roughly  from  the  micro  to  the  macro  level.  First,  table  9 gives 
the  intercor relation  matrix  of  standardized  test  scales  for 
each  issue.  Correlations  of  less  than  .70 — pairs  where  less  than 
half  of  the  variance  is  accounted  for  --  are  highlighted. 

In  each  issue,  most  of  the  low  correlations  are  due  to  a few 
participants  whose  perceptions  differ  considerably  from  those 
of  LOG  analysts  and  their  fellow  t-est  participants.  These  out- 
liers largely  account  for  the  lower  reliability  figures  for 
test  participants  reported  in  table  10,  where  LOS  analysts' 
tests  are  evaluated  separately. 


^^Torgerson,  p.  69, 
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TABLE  8 : SCALING  CRITERIA 


CRITERION 

POLICY  LIKITS 

degree  of  restriction 

No  restrictions 

imposed  on  passage  of 
vessels  through 
straits 

1 

Straits  closed 

degree  of  control  over 

seabed  mining  to  be 

No  control 

accorded  any  new' inter- 
national seabed  agency. 

1 

Complete  control 

degree  of  coastal  state 

control  over  research 

No  control 

conducted  in  the  coastal 
zone . 

1 

Complete  control 

TABLE  : INTERCORRELA.TION  MATRICES,  STANDARDIZED 

SCALING  TESTS,  BY  ISSUE 


TABLE  10:  INTRACLASS.  CORRELATIONS 


a 


LOS 

TEST 

Issue 

^3 

"l 

^3 

^10 

Seabeds 

.93 

.98 

.76 

.90 

.97 

Straits 

.95 

.98 

.71 

.88 

.96 

Science 

.97 

.99 

.75 

.90 

.97 

^ cov  ^ kri 

vaT  ' 1+  (k-1)  r • 

B.  J.  Winer,  Statistical  Principles  in 
Experimental  Design,  2nd  edition  (New  York: 
McGraw-Hill  Book  Company),  pp,  291-296. 
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The  reliability  of  a single  measurement  (r^^)  averaged  .95 
for  LOS  analysts  and  .74  for  other  test  participants.  The 
reliability  of  the  mean  of  all  ten  test  participants'  results 
(Tiq)  approaches  that  of  LOS  project  members.  The  consistency 
of  the  results  suggests  that  a much  larger  sample  of  the 
policy  community  at  large  is  required  to  produce  a policy  scale 
as  reliable  as  that  produced  by  a highly  socialized  group.  More 
interesting  than  this  commonsensical  finding,  however,  is  the 
reported  reliability  of  the  mean  of  three  measurements  for  test 
participants,  which  averages  .89  with  almost  no  variation  over 
the  three  issues.  That  is,  the  reliability  of  a policy  scale 
based  on  a sample  of  three  outside  experts,  while  not  perfect, 
can  be  expected  to  be  sufficiently  high  to  be  used  as  a framework 
for  analysis. 
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In  practic'e,  the  project's  approach  to  the  problem  of 

diftermg  percept i (ins  has  been  to  rely  on  a collegial  scaling 

process  that  takes  into  account  several  points  of  view.  To 

simulate  this  process  in  the  test,  we  developed  issue-var tables 

for  each  issue  based  on  the  composite  test  results  of  LOS 

analysts  and  other  test  participants,  respectively.  Following 

Torgerson,  the  mean  scale  value  assigned  to  a given  theme  in  the 

tests  (standardized  for  comparability  across  participants)  was 

17 

taken  as  its  scale  value  on  the  composite  issue-variable . The 
resulting  composite  scales  were  then  transformed  monotonically 
to  produce  a scale  approximately  25  units  in  length  with  an 
origin  of  1.  The  resulting  variables  are  shown  in  table  11. 

Table  12  gives  the  results  of  a comparative  analysis  of 
the  composite  variables,  starting  with  the  comparative  rank-ordering 
of  themes.  The  respective  Spearman  rank-difference  correlations 
(Spearman’s  rho)  are  .97  for  seabeds,  .99  for  straits,  and  .87 
for  science.  T)iough  LOS  analysts  and  test  participants  differ 
most  on  the  rank-ordering  of  science  themes,  test  participants 
differ  least  among  themselves  on  that  issue,  as  shown  by  the 
standard  deviations  of  ranks  within  themes,  across  participants. 

A standard  deviation  of  ♦■/-  2.0  scale  values  indicated  substantively 
significant  differences  of  opinion  among  participants  as  to 
the  proper  ranking  of  a given  theme.  Such  difference.*!  occurred 
over  the  ran),  ing  of  t’ne  proposed  regime  of  coastal  state 
consent  for  "resource-related"  research,  and  over  the 
question  of  coastal  state  control  over  the  publication  of 

^ ^Torgerson,  p.  71. 
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COMPARISON  OF  COMPOSITE  VARIABLES,  SCALING  TEST 


! 

I 
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Substantively  significant  differences 


research  results.  With  the  exception  of  the  latter  theme, 
participants  tended  to  disagree  least  on  themes  ranked  near 
either  end  of  the  respective  composite  scales.  That  is,  as 
one  might  expect,  participants  were  able  to  place  more  extreme 
proposals  with  greater  consistency.  Even  so,  on  average , parti- 
cipants' rankings  of  even  the  more  ambiguous  themes  tracked 
closely  with  LOS  analysts'  rankings. 

To  compare  the  policy-spaced  composite  variables,  test 
variables  were  regressed  on  their  LOS  project  counterparts. 

The  results  are  reported  in  the  last  column  of  table  12,  and 
plotted  in  figures  1 through  3.  In  each  regression,  the  LOS 
composite  accounted  for  at  least  85%  of  the  variance  in  the 
test  variable.  Thus,  independent  measures  of  perceptions  for  each 
issue,  drawing  on  two  different  groups  of  substantive  experts  — 
one  heavily  socialized  in  the  particular  methodology  being  tested 
and  one  not  socialized  at  all  --  produced  results  that  were  strik- 
ingly similar  across  groups. 

Because  the  regressions  were  run  using  non-standardized 
scales,  the  residuals  can  be  directly  interpreted  in  terms  of 
scale  values.  Themes  with  residuals  of  three  or  greater  are 
labeled  in  figures  1 through  3.  A positive  residual  means  that 
test  participants  considered  the  proposal  embodied  in  a particu- 
lar theme  to  be  more  restrictive  (of  access  to  the  seabeds; 
freedom  of  transit;  or  freedom  of  research)  than  did  LOS  analysts. 
A negative  residual  indicates  a less  restrictive  interpretation. 


FIG.  1:  C WPOSI'PE  VARIABLES  REGRESSION:  SEABEDS 


J.^:3K03j  Sc?  7 


High  positive  residuals  tended  to  be  associated  with  "umbrella" 
proposals  that  seem  to  leave  operational  discretion  to  the  coastal 
state  (or  in  the  case  of  the  deep  seabeds,  to  the  international 
agency) . Thus,  more  restrictive  interpretations  were  given 
the"Enterprise"  system,  "exclusive  riparian  state  jurisdiction" 
over  straits  transit,  "coastal  state  control"  over  scientific 
research,  as  well  as  a proposal  for  "reasonable"  coastal  state 
regulations  for  research.  Test  participants  tended  to  take 
more  of  a worst-case  approach  to  such  proposals  than  did  LOS 
analysts.  A similarly  consistent  pattern  is  not  evident  for 
high  negative  residuals. 

The  respective  composites  produce  very  similar  results 

when  used  to  create  distributions  of  countries'  preferred 

1 8 

positions  (figure  4).  The  issue  where  LOS  and  test  perceptions 
vary  the  most — scientific  research — shows  distributions  of  data 
and  estimates  that  nevertheless  closely  parallel  one  another. 

The  respective  medians  for  science  are  substantively  identical 
(coastal  consent  for  research) . 


Interpretation  of  the  histograms  is  straightforward. 

Scale  values  form  the  horizontal  axis  for  each,  while  the  verti- 
cal axis  represents  number  of  countries.  Thus  the  height  of 
the  column  for  a given  scale  value  gives  the  number  of  countries 
with  preferred  positions  at  that  value. 

The  distributions  in  figure  4 use  national  position  data 
drawn  from  UN  documents  for  the  Caracas  and  Geneva  sessions  of  the 
LOS  Conference  (1974-75) . Because  they  are  based  upon  the  12-theme 
test  issue- variables , they  do  not  necessarily  match  project  analyses 
for  1974-75  based  on  more  complete  scales,  and  are  valid  only  for 
comparing  test  groups,  within  issues. 
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C'jJ-'.l'AkA  U Ve.  Dir/rRIBUTIONS  OF 
COMPOSll'h  I3of)F-'7ARrABLES 


Seabeds:  LOS 


Countries  whose  preferred 
positions  are  based  on  a 
national  score  & a 
regression  estimate. 


Science:  Tost 


(To  which  are  added:  ) 


Countries  with  preferred 
positions  based  on  regres 
sion  estimates  only. 


These  results  demonstrate  the  relative  robustness  of  the  project's 
policy  scales,  and  their  relative  insensitivity  to  minor  differ- 
ences in  perceptions  of  the  issues.  The  chief  factor  determining 
the  overall  effect  of  any  change  in  ordering  or  policy-spacing 
is  of  course  the  proportion  of  data  accounted  for  by  reordered 
themes.  The  larger  the  proportion,  the  more  serious  the  potential 
impact  of  reordering  on  analysis.  Thus,  even  though  the  policy 
scales  themselves  are  rather  robust,  it  is  important  that  pro- 
cedures for  creating  them  be  reliable.  The  results  of  this  test 
suggest  that  they  are. 

A VALIDITY  CHECK:  LOS  AND  THE  SNT 

The  results  of  the  scaling  test  suggest  the  validity  of 
the  LOS  project's  models  of  the  law  of  the  sea  negotiations,  at 
least  for  the  questions  of  the  deep  seabeds,  scientific  research, 
and  straits  transxt.  A further  validation  effort  has  been  under- 
taken. Although  the  Single  Negotiating  Text  (SNT)  issued  at 
the  Geneva  (spring  1975)  session  of  the  LOS  Conference  is  not 
the  final  treaty,  it  does  represent  the  considered  judgments 
of  the  chairmen  of  the  three  main  committees  of  the  conference 
as  to  the  direction  of  negotiations  in  their  respective  commit- 
tees at  that  time.  If  the  SNT  did  not  represent  the  "true" 
preferences  of  Conference  participants,  it  was  used  as  the 
point  of  departure  for  the  next  round  of  negotiations.  Compar- 
ing LOS  project  medians  on  key  issues,  generated  just  prior  to 
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Geneva,  with  the  relevant  provisions  of  the  SNT  wi 1 1 art:  least 
tell  ns  whether  the  project's  forecasts  were  at  all  "in  the 
ballpark. " 

Provisions  of  the  SNT  that  corresponded  to  our  major 
issue-variables  were  coded,  in  the  sense  that  appropriate  scale 
values  were  assigned  to  them.  These  were  compared  to  the  medians 
of  the  corresponding  issue-variables.  Table  13  gives  these 
comparisons.  Differences  are  presented  as  a range  where  the 
complexity  of  the  text,  or  conflicting  text  articles  relating 
to  the  same  issue  made  assignment  of  a single  scale  value  to 
the  text  impossible. 

According  to  the  criteria  we  have  been  using  in  this  paper, 
no  median  was  seriously  off  the  mark-  No  difference  was  more  than 
three  scale  values,  according  to  our  evaluation  of  the  SNT. 
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SUMMARY  AND  CONCLUSTONS 


This  paper  has  reported  the  results  of  reliability  and 
validity  tests  run  on  the  CNA  Law  of  the  Sea  Project's  thematic 
content  analysis  and  pol icy-scal inq  procedures.  The  results  of 
the  codinq  test  suqqest  that  the  content  analysis  procedures 
are  very  reliable,  even  with  marqinally  tutored  coders;  that 
substantial  socialization  raises  codinq  reliability,  while 
decreasinq  cultural  bias.  Errors  of  omission  remain  a problem 
for  complex  issues,  but  experienced  coders  can  be  expected  to 

pick  up  85%  of  codable  remarks  on  such  issues,  while  coding 
relatively  few  extraneous  remarks,  prior  to  quality  checking. 

With  quality  checking  procedures  in  use  as  described  in  the 
introduction  to  this  paper,  error  rates  in  the  final  data  base 
used  for  analysis  should  be  lower. 

Although  perceptions  of  the  issues  under  negotiation 
vary  even  among  the  experts,  collegial  policy-spacing  as 
practiced  by  the  LOS  project  can  be  expected  to  produce  issue- 
variables  that  are  on  the  order  of  90%  reliable  and 
fairly  robust.  A comparison  of  LOS  project  forecasts  with  the 
1975  Single  Negotiating  Text  suggests  further  that  issue- 
variables  so  developed  can  be  valid  indicators  of  trends  in 
negotiations. 
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This  information  processing  system  and  its  attendant  ana- 
lytic and  forecasting  models  are  applicable  not  just  to  the  law 
of  the  sea.  They  can  be  applied  equally  well  to  other  multi- 
lateral negotiations,  past  or  future,  for  purposes  of  trend  and 
outcome  forecasting,  or  historical  reconstruction.  Where  there 
are  reports  of  countries'  negotiating  positions,  they  can  be 
coded.  Where  the  policy  dimensions  underlying  the  issues  are 
known,  or  can  be  deduced,  issue-variables  can  be  created.  These 
procedures  are  not  without  cost.  Content  analysis  in  particular 
is  a laborious  process.  Coders,  both  experienced  and  inexperi- 
enced, took  an  average  of  two  hours  to  complete  the  coding  test, 
generating  about  60  items  of  information  apiece.  If  this  rate 

is  typical  of  the  entire  LOS  data  base,  then  its  creation  required, 
over  a period  of  years,  about  1,500  hours  of  coding.  The  pay- 
offs, however,  especially  in  a large  and  technically  complex 
negotiation,  can  be  substantial,  giving  negotiators  systematic 
information  about  the  shape  and  progress  of  negotiations; 
a macroscopic  counterpoint  to  the  day-to-day  interactions  of 
diplomatic  bargaining. 
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TABLE  A-1 


E^-JDEPENDENT  VARIABLES  USED  IN  1,0S  PROvJECT  KELKESS10N3 


and  Geographic  Variables 

Offshore  Hydrocarbons 
Major  Oil  Producers 


Major  Mineral  Producers 


Distant  Fishing  States 

Major  Fishing  States 

Landlocked 
Shelf locked 
Narrow  Shelf 
Broad  Shelf 

States  on  Semi-Enclosed  Seas 
Straits  States 


Blue  Water  Navy 
Coastal  Navy 

Major  Merchant  Fleet 


Sources 

Summary  1972  Oil  and  Gas  Statistics, 
for  Onshore  and  Offshore  Areas  for 
151  Countries,  USGS  Professional 
Paper  885,  1974  . 

Summary  Petroleum  and  Selected 
Mineral  Statistics  for  120  Countries, 
USGS  PP  8)7,  1973. 

Department  of  State  Estimates,  1972. 


Limits  in  the  Seas,  Department  of 
State  International  Boundary  Study, 
Series  A,  No.  46,  1972. 


Sovereignty  of  the  Sea,  Department 
of  State  Geographic  Bulletin  No.  3, 
Revised  October  1969. 

Jane's  Fighting  Ships,  1973-1974 


Lloyd's  Register  of  Shipping,  1973 


UN  Statistical  Yearbook,  1972. 


POLITICAL  VARIABLES 


Regional  Groups: 

Af r ica 
As  i a 

Latin  America 
Eastern  Europe 
Western  Europe 
and  others 


Scandanavian  Group 
U.S.  Treaty  Allies 
Caribbean  Bloc 
OPEC 

Archipelago  States 
European  Economic  Community 


Group  of  77 
Arab  League 
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TABLE  A-2:  ISIBUE-VARIABLES  USED  TO  CALCULATE  NATIONAL 
SCORES  FOR  CODING  TEST 

VARIABLE  1:  TERRITORIAL  SEA  DELIMITATION 


SCALE  THEME 

VALUES  THEMES  NUMBERS 

1 LL  OPPCSf^  EXTEr^SICK  JF  CQ4LTAL  STATES  CLAIMS  7C 

1 10  MICE  TEE.kI  TORI  AL  3E^  SO  q 

1 6 MILE  TERSITO'^IAL  SEA  713 

1 A MILE  TtKEITORIAL  SEA  .....  723 

1 modify  positive  law  rlY  AGPEEMENT  ONLY  ^25 

1 FOR  NARROW  TLKttITORIAL  SEA  955 

3 FOR  12  MILE  T E F=’f  T Dr.I  a'l  SEA  934 

4 MITE  TAKE  1 2M I T3/20C  EZ  WFREENAV  ♦OVERFLIGHTS  7l 

4 , 12MILE  T:  TIED  To  EXCLUSIVE  ECONOMIC  ZONE  334 

4 TS  DELIM  linked  TO  FREEDOM  OF  PASSAGE  THRU  aTRAT 921 

4 SYMPATHETIC  TO  PA CK A GE 1 1 EM  I LE  T3, SPECIAL  CONT  Z) 322 

4 MIGHT  TAKE  12  965 

4 DISCUSS  12  Mi  TS  IF  NO  RIDERS  1028 

5 NEED  UMFG=M  TERRITORIAL  SEA  927 

6 12MILE  FOR  CEA/FARTHtR  FoR  OCEAN  965 

7 Id  MIlE  TER'RITi'F.I  Al  SEA  704 

7 15  MILE  TEP.RITCRIAL  SEA  707 

9  FOR  25  MILE  TS  39  0 

9 FOP  JO  MILE  TS  992 

9 50  MILE  TERRITORIAL  SEA  2 

10  T3  OFFINES  ON  BASIS  OF  RECIPROCITY  W/NEIGH9JRS  321 

10  T3  DELIMITATION  DIFFERENT  IN  NARROW  SEA, OPEN  SEA 931 

10  FIND  Flexible  criteria  fcf  tl  delimitation  935 

11  FOR  100  MILE  TLFFITORIAL  SEA  300 

12  130  mile  territorial  SEA  999 

IJ  150  MILE  T3  .....  Ill 

15  MIGHT  GIVE  UP  200  MILE  TEF:.R1T  ORIAL  SEA  237 

16  SUPPORT  LA  CLAIMS  OF  200  MI  TS  39 

17  FOk  aTATE  PIGHT  TO  DELIMIT  SOVEpFignTTY  ZONE  305 

17  SYHP-THPTiC  7C  Uf.'lLATERAL  CL-iI»3,THO  NOT  CONE  CC 314 

17  TS  NEEL  regime  PlUGALITV  0!-  vEGION  tl^SIS  FOR  336 

17  plurality  HILL  NOT  LlAl  TO  ANAKCHY  93ft 

17  retain  P'^E-9ATIF  T3  GRTR  Than  PPESCRIBED  HY  CONV 1237 

19  CPPCScL  TO  12  NILE  TEFFITCFIAL  SEA  .....  963 

20  LA  H>-S  SPECIAL  SLCUKITY  REQUIREMENTS  907 

23  200  MILL  T^' Tr  ITORIrtL  SEA  908 

20  BIO.Mm  THcOpy-CCmST  ♦WATER  IS  r;ATURAL  GEOGRAPHY  311 

20  OE'VElOFEQ  wish  To  restrict  JUxISZUNIL  AT  CLAIMS  316 

20  200-MI  DL*,IM  GOOD  MODEL  FOP  LDC  317 

23  BROAD  NAIl  JURISU  PROTECTS  AGAINST  PRIVATE  DEV  318 

20  TS  ClNCEPT  was  BASED  Cl^  MILITARY. NOW  ECON  + EOQL  957 

20  200  mile  has  IDEGLCGICwL  SIGNIFICANCE  1307 
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VAKlABLi;  2;  C'jASTAL  ZONL  SC  1 liN'TI  I- 1 T PI. SEARCH 


SCALE  THEME 

VALUES  THEMES  NUMBERS 


I NO  C-,  - EST?  IC  riONS  O'i  BASIC  PES'A^CH  BEYOND  T3  375 

t maintain  fA,‘;eOoM  CF  OCEAN  SCItKOE  120 

1 SK  In  EZ  i;F  SES^PtONu  AOPmTj  NOT  »^ITh5TAN0C>  •••<•  1A31 

2 SR  SUB.jrCT  TO  t-OLL  FESTPICTIONS  IN  ECONOMIC  ZONE 372 

2 SR  IN  fcCON  ZCNE  COMPLY  W/If.TL  POLL  STNOS  1103 

T NO  CuNSEfiT  F(  ^ f.*.E  -<cLATFO  SP  IN  07  1499 

i C0  4ScNT  kfcf,^-E  INCR  CCST/JECR  y/OL'JME  OF  PESZARCH 1111 

ACCESS  TO  INFO  IN  RETUFN  FCR  FREEPUM  OF  SCIENCE  338 

4 INTL  GUUELINE„  F0\  SK  IN  F.Z«-5HEcF  879 

5 CONSiOEF,  NGTIF  to  CS  FOR  SCI  FES  O^ER  SHELF  401 

b NOTIFY  RES  EZ  r F.  S3  RESOURCES  + COASTAL  FISH  329 

b SYMPATHFTIC  TO  US  SCI  RES  >LRTICLES  1055 

7 FOR  reasonable  CS  RECS  FjR  SR  IN  ECONOMIC  ZONE  377 

7 SPECIFIC/LIMT 0 CS  RIGHTS  OYER  RFS/POLL  IN  EZ  17 

7 NOTIF  TO  CL  FOR  SR  IN  EZ  338 

7 kESFARohEk  YtSTEO  W / C 6 L I GA  T 10  N/SH  a R I N G 51 

7 ALLCH  NOTIF  TO  CS  FQP  SCI  RES  OYk  3h£lF  399 

7 CONSENT  FOr.  SR  IN  EZ  ASSUMED  IN  FIxF  TIMEF’ERIOD  414 

3 SET  TYPE  RES  DONE  FREELY  ♦ THAT  \EQUIRGG  CS  CONS 191 

8 CS  ClNSNT, FE  RfLATEO  SR  IN  EZ, BASIC  SR  FREE  764 

3 FREE  CF  IN  NC/CS  CONSENT  IF  CHANGE  SI  STIUCTURE  268 

3 CS  CCNSEGT  F 3^  OVR  SHELF  IF  STRUCT  OF  Sh£LF  CHG 373 

9 CS  CONSENT  FCR  SR  IN  EZ  - FALLBACK  327 

9 CS  GIVE  PREF  ThTMT  TO  NEIGH  LL/SOS  S’R  a>^.OUESTS  950 

9 COfJC  SUPPORT  CCNSENT  REGIME  FOR  F*E-RELAT£0  CR  1146 

10  CS  FITE  TO  SU5FENC  RES  IN  EZ  IF  COf^OS  VICLATJ  160 

10  CS  COnSNT  F/USE/PLACEMNT  of  MONITOR  EOUl=  1 .9  EZ  1397 

III  S=>  CONSNT  kF(jm:  OESIGNO  to  lever  tech  transfer  1448 

IJ  CS  CCNELNI  (lZ  Sr-I,  BUT  GIVEN  IF  SCfOITUIONl  MET 1369 

10  CONSENT  FOk  oR  IN  Z HUT  NOT  UN  JU  5 7 1 I A IL  v ,'hEi.O 1145 

10  FOR  oS  CONSEfiT  REGIME,  RUT  RILLING  TO  BARGAIN  618 

11  CS  Pr  TO  SUjF  SR  w/CONCUFRFNCE  CF  APPRO=  UN  OOCY 1136 

12  CS  -'ITI  TO  CL  T E + F-RELATC  FES-CASE  lY  CASE  151 

12  pollution  necessitates  CuASTmL  STATE  ,Rf  S CONTROL .315 

12  CS  CCnSFNT  <L0  for  SR  CVk  SHELF  326 

12  SCin.TIFlC  PfCfARCH  N/CONSFNT  of  coastal  state  302 

12  REG.NL  AOaMTJ  AHNu  nlHAr.IAN  STS  FOR  SR  IN  S''Z  1432 

13  NON-JUPL  1C  able  S->  specimens  REMAI.N  PRCPTY  CS  1437 

13  CS  ouiJL  CCiZtNT  ( SR  > BUT  NOT  UNNECESSLY  N/HlLP  1438 

13  ntJECT  P«10r  NGTIFN  FCr.  Sk-  INAOEO  PROTN  PC'  CS  1121 

14  coastal  3TAT1S  HAVE  RIGHT  TO  CONTROL  RL.IEA-Ch  314 

14  CS  CLuTPOU  Sr>  CN  CO.iTIl. ENTfL  SHElF  .3  79 

15  CS  IS  OWhEP  lF  (jaTA*  sample  Fn.M  S'v  IN  f.  7 ,,,,,  245 

15  COAS’'-L  ^I^Tc  lChTpJL  SCIEI-CE  20)mtLE3  335 

15  OS  KT  TO  SUPFrt.ES  SR  C A T A/ C OnT  ROl  pUClICATICn  233 

15  EAUUkl  TO  -.LF’I  Y INOIC  CS  fE'FL'SAL  OF  SR  RE.CUfST  771 
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TABLE  A-2:  continued 

VARIABLE  3;  IvliO  WILL  EXPLOIT  THE  INTERNATIONAL  AREA 

SCALE  THEME 

VALUES  THEMES  NUMBERS 

1 REGI  jTkY/FI T CO^IE  FIF.ST  -E-VE3  711 

2 NO  OIRfcCT  Exc'LC  IT  ATION  tlY  KA3Hl';L'OY  719 

4 CAN  TRk.NSF*-  LIiENS  ^IGPTS  R/Q  ttUTnO'^.TYS  CONSENT  177 

4 OPPOSE  QUcTA  SYSTEM  IN  ISRA  AREA  403 

•4  REGUlwTGHY  FLR  E*E  -TEYLNu  NATL  JUKISJICTICN  712 

4 INTL  licensing  SYSTEI  717 

4 licenses  OASLC  in  FKEL  grid  system  722 

4 OOUrlLt  COfjCESSION  SYSTlM  IN  “EGIME  7G1 

5 rkELlNaul^Hy£t.T  RECuI<EL  CF  SUBLICENSEE  ONLY  439 

5 .GT.  1/?  rELlNU  -ET  E V».L,CC  NTRACT  R CHOOSE  A-EAS  G24 

b QUOTm  UiNuEk  licensing  system  in  IS-;A  AREA  402 

b KELlNQ  ULJ:  In  jeep  SB  IF  NC  t*E  W/IN  TIME  <^RamE 4A3 

b equal  ^EEAu  GJCTAS.ALL  STS. NO  POP/SIZE  CRITERIA  A33 

b FOR  kELINO'.J1S«HENT  IN  DEEP  SB  AREA  G79 

6 CONTRCTR  RE -INN  1/3  OF  ArEA  BEFO^'E  9EGNG  EX°LT  1420 

7 P.ESC'JRCE  M^NASkMENT  COMMISSION  175 

7 flexible  on  ISr.A  LICENSING  SYSTEM  45b 

B RESEr.vE  PERCrNTAGE  OF  EACH  OCEAN  FOR  COMMON  HE^T 720 

a LEASING  B»S£0  CN  P jP'JLATION  ANO  GNR  7?  i 

6 RESERy/L  BLuC  FCR  FUTURE  EXPLOITATION  543 

5 AUTH  TO  Cu:.:fOl  transfer  CF  LICENSE  IN  iNTL  ZONE 1045 

0 MIXEU  MAy^HINCRY  AS  FALLBmCK  FROM  LICENSING  ?7b 

1 SET  UP  SK-LETCf,  M ACH  i Nt  R Y , £ XP  AN  0 LATER  751 

9 US  BANKING  P'-CPOSAL  1531 

10  MiXEu  MACPl>iLRY-  UIRECT  ANf  LEASE  E +E  181 

10  assembly  GO/^P^  E + E OIFECTLY,  JOINT  OR  LICENCE  S55 

11  uRGAmS  ACCORuIr.G  TO  TANZAf.IAN  ;iOQEL  773 

12  .MIGHT  ACCEPT  JLltJT  ViNTL^-ES  ISRA  AREA  .....  113 

li  INTL  JRERATING  mGEiJCY  FVENTUALLY  NOT  NON  725 

13  inclined  TCMhRD  INTL  CPE-.ATING  agency  769 

13  PROCU  SYS  ISRA  CHCICE-PHASf  OUT  LiC, PHASE  IN  OIR 805 

14  SUPRT  CONTRmCTUAl  JOINT  \/FfTUiRE  QVR  EOUIIY  PARTI 1593 

15  INTL  operating  ASENCT  UIThIN  UN  SYSTEM  715 

15  CREATE  INIL  Sl»-BlC  COrPCK.ATIQN  729 

15  EST.ABLISH  JOINT  E N TERPP  I S£  S 7‘il 

15  JOINT  FSr.C  MGT  ESSENTIAL  PAkT  CF  COMMON  HERITAGE 837 

15  CONTPaCT/J\/  W/ST3. firms-  If.  TIME  DIRECT  E*E  ONLY 839 

15  SB  RSRCS  IM  LITU  REPRFSNT  LDC/AUTH  CONTRI9  TO  JV 877 

lb  ENTErRhlSE  SYSTEM  18b 

lb  LICENSING  FNCAr.GcRS  MANY  STATE  INTERESTS  795 

lb  OPPOSE  MIXS  MACH-UNFaIK  COMPETN  FRCM  licensees  815 

lb  LIC  INL0MP<,TP.L  W/COM  k^ER, obsolete  ,P«TERMALISTIC  8b0 

lb  LiCEf.SING  k.'IClEr  F I.NANCL /TECHL  GAP  3ETW  UC/lDC  1125 

17  OIKHCT  EXPLOITATICN  JY  INTL  AU7HTY  1022 
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l)ie  InternaDonai  Studs  Assu  lation  Meronw.  27  I eh 
19761  AD  A0:2  591 

PI’  150 

Kelly  Anne  M and  JVtersen.  f fuirles  "Retenf 
Changes  in  Soviet  Nava)  Pob<  y Poopects  (or  Arms 
(imitations  m the  Medifertanean  and  Indian  rXean." 
28  pp  Apr  1 976,  AD  A 023  72  3 

Pf  151 

tlor«)wiie.  Stanley  A 'The  hconoiiiK  < onsequcnccs 
ol  Political  Philosophy."  M pp  . Apt  1976  1 kcprinud 
Ifom  IcoiuxiiK  Inquiry.  Vol  \IV  N«i  I.Miu  I97M 

PP  152 

Ml/  ihi.  Maunce  M . "fin  Path  Integral  S»>lunoris  of 
tlkC  Sch'odingei  I quation.  Without  limiting  Pri> 
Leduie."*  lOpp.Api  1976  tRepnntcd  from  Journal 
ol  MathcmaiKal  Pliyscs,  Voi  W.  Nrv  4 (Apt  1976), 
*^66  575) 

*Ri'utifth  jui>P’>rtrJ  by  ihc  .\aiional  Scuncr 
FounJafn/n 

f*P  1 5.3 

Mi/rahi,  Mauiuc  M , "W'KB  I Kpansions  by  Path 
(ruegraly  W'lth  Applications  to  the  Anharmoitk 
Oscilliioi. '*  137  pp,,  May  1976  (Suhiiuttcd  !t*r 
publication  in  Annals  of  Pliyui si  AD  A025  44<) 
•Rere<ifr6  lup/wm-J  6i’  (He  Sirrmr 

hiundatvm 

PP  1 54 

Mi/ialii  M-umcr  M "On  the  Sciiu  t lassital  I a 
pansion  in  fjuanlunt  Mechanics  for  Aibitiary 
ilamiItonian!i."  19  pp  , May  1976  (To  appear  in  the 
Journal  of  Marnematical  Physics)  AD  A025  441 

IM’  155 

Squires.  Mk)ucl  1 "Soviei  J'ortngn  Policy  and  Thud 
World  Nations."  26  pp  , Jun  1976  (Prepared  lor 
presentation  at  the  Midivest  PohtKaJ  Science  As-umj 
tion  meetings  Apr  30,  1976)  AD  A02H  388 

IT  1 56 

Srslfings  W'lllum.  "Apprr»ac!iei  to  Chmese  Cha/actef 
Kecognition.'*  12  pp  Jun  1976  (Hepnnied  trotn 
Pattern  Ke<.iignitlon  tPtrgainofi  Pressl.  V'lil  8 pp 
87-9h  1976)  AD  A02H  69? 

IT  15  7 

Morf^n.  W’lihain  F , "f inemploymcni  and  Kic 
Pentagon  Budget  Is  TJierc  Anything  ui  the  I inpty 
Pork  Harrer  ' 20  pp  . Aug  19/6  AD  A030  4^5 

IT  158 

Haskell,  l.Cdf  Rtchaid  I)  tUSNi  "Fapenmeiital 
Validalioti  of  Piobahtlity  Piedictioiu,"  25  pp  Aug 
|9'’6  (Presented  at  the  Military  Operations  Keseaich 
S«KletV  Mrriiiig.  I- all  1976)  AD  A030  458 

PP  159 

McConnel/.  Jame.s  M "The  DorshJuiv  A/deJes.  Die 
New  (ioishkov  Book  and  Then  Relation  (o  Poiky," 
93  pp  . Jul  /976  (To  be  printed  tn  Vr**-//.*  Varw) 
Influence  fkrmejric  ami  Ftfeijpi  Dimentuim.  cd  by 
M Mcefisvire  and  J Mi  Donnell  New  York  Praegrr) 
AD  A029  227 

IM’  IfiO 

WiIm^.  iXttTMmd  P . Jr  . "ITse  U.S  Sixth  f ieet  and 
the  Convenhonal  IXfense  of  lutojie,"  50  pp  , Sep 
1976  (Siihmiiied  lor  pubikalion  tn  Ailrlphi  Papers. 
IISS.Iondon)ADA0.i0  457 


PP  !6l 

Melich,  Muhaet  i anu  Peer,  Vice  Adin  HavtINS 
Retired),  '(ieet  f •.mmanders  AHoat  (M  As/ro'e’"  9 
pp  , Aug  19*^6  (Reprinted  from  V S Sjval  Imfitufr 
Hnu-eeiUng\  Inn  1976)  Al>  5030  456 

PJ’  162 

Fnedheim  Robert  1.  . "Parliameniiry  [)ipl<iiMai> 
106  pp  Sep  J9‘*6  AD  A033  306 

PI'  1 63 

Ukkman,  Robert  F . "A  Model  for  PredkUng  Re 
cruit  losses."  9 pp  . Sep  1976 (Presented  at  f/ie  84rh 
annual  convention  of  the  Ameikan  Psychologkal 
Association,  Washington,  DC  , 4 Sep  1976) 

AD  AO 30  459 

I’l’  U4 

Malioncy,  Ruben  h . Jr  , "An  Aiaessioerii  ol  PiiMk 
and  FTiic  Perceptions  in  FratKe,  TJir  llrutrd  king 
dom.  and  the  Federal  Rrpublk  ot  (iermanv  31  pp  . 
Feb  1977  (Piescntcd  at  Conlerence  "Pet>.epU»m  o( 
the  U.S  Soviet  Balanic  and  the  Politkal  I Vs  cd 
.Mtliiary  Power"  '^>ns<‘(ed  by  Director.  Advarued 
Research  PriMe  .#ncv  April  1976i  aD  036  59*r 

PP  165 

Jondrr»w,  lames  M "Fflei.tso/  I..KJe  Keirrhn»>r»  <*i 
Imports  of  Steel."  67  pp,.  N >*«nibef  I9‘'6  (lx 
livered  at  fI.AB  ( onference  in  Dtv  I97ft) 

PP  166 

Feldnun.  Paul.  "Impediments  to  the  IniplemenUdon 
of  Osiisble  rtungrt  ui  the  Regulation  of  Urhar 
Public  Transportation."  12  pp  (Xr  19’’6 
AD  A033  322 

PI’  166  Revised 

Feldman  Paul.  "WTiv  It's  UttOcult  to  ( hange  Krguia 
*>«n."  fXt  1976 

IT  167 

Klemman.  Sarruiel,  "Rf)T('  Servke  ri’uumtoxrjts  a 
Comment."  4 pp  , Nov  1976.  (To  he  puMisbed  if' 
Public  Choice,  Vol  XXIV.  Fall  1976)  AD  A031  *05 

PP  168 

Lockman.  Robert  F . "Revabdaiion  ol  CNASuppi>rt 
Personnel  Selection  Measures."  36  pp  . Ns>v  I9t6 

PP  169 

Jacohson.  Louis  S . "Fartnngs  liisars  ol  Wotkm 
lAsplaied  Irom  Muiutactunng  Induslnet,"  38  pp  . 
Nov  1976.  (Delivered  it  II  AB  Conference  in  IXs 
1976) 

IT  170 

Hrechimg,  I rank  P.  "A  Time  Srrjc  Analysis  of 
leboi  Tuinover,"  Nov  1976  (Deli>cied  at  ilAB 
Confererk-e  in  Dec  1976) 

PP  171 

Ralston  James  M . "A  Dittustoi  Mivkl  fot  (aP  Red 
I FD  Ixgrtdation.  ’ 10  pp  . Nov  I9?6.  (Kihlislied  in 
Journal  ol  Applied  Pysici.  Vol  47.  qp  451K4527. 
Oct  1976) 

PP  172 

Clasaen.  Kathleen  P , "Unemploymem  irsunnee  and 
the  length  of  Unemploynwiil."  1)bc  1 976. (Presented 
at  the  (''iiveisitv  of  Rixlirsier  labor  W.irkshopon  16 
Nov  1976) 

PP  17.3 

Klemman.  Samuel  D . "A  Nitte  on  Racial  DMieieniev 
m the  Added  Wotker/DnciHinafed  Wmkff  fontro 
vffsy,"  2 pp  , Dec  1976.  (Published  in  the  A'uemen 
fcwnomisi.  Vol  XX.  No  I.  Spring  1976) 


CNA  Professional  Papers  — 1973  to  Present  (Continued) 


IV  P4 

MAhnrxN’  Roh«n  H , Ji  A ('<>mp4n«<tn  oi  the 
B(<><>iun|{>  *r>d  InictnitimidJ  Incidents  Pr<>)r<.ii.“  12 
pp  Fet.  o**?  AP0J7  21)6 

PH  175 

le^inc  D*mel.  Sioloft,  P«t<i  wd  S|kiji11.  N*n  y, 
*Pubii>  liru|  Tiettniem  and  Addle'  <’iune,’*  iune 
1^76,  (Ptihlished  in  lournti  ut  I < toil  Studies.  V'<il  5, 
N(  2) 

PH  176 

Icux  Weruli.  ‘('oireljies  >it  Retention  and  Pr»n><>- 
tum  fi'i  I SNA  dfiduafei."  3f  pp  Ma;  1^*77 

HP  l77 

loJunan  Robert  p and  Wamei.  i«>hn  7 Predic- 
ting Attiitton  A Test  ol  Alternative  Approaches.*'  13 
pp  Mai  1977  tPresenied  at  the  DSU/ONK  Con 
'ereiKC  on  InUited  Attrition  Xerox  International 
Tralnmg  Center,  leesburg.  Virpma,  4 7 April  l<>7'») 

PP  |7« 

Kteinnian.  Samuel  O *'An  Evaluation  of  Navy 
Inreatrtited  lane  Officer  Accession  Piogranu"  23 
pp  Apnl  1977  (Tc  be  presented  at  the  NAft) 
( inference  on  Man(k  %ver  narumig  am)  (HgaMtaaii'in 
tieafn  Sneia,  haiy.  20  June  1977| 

PH  J79 

Stolof!  Peter  H and  Balui.  Stephen  J . “Vacate  A 
M.>del  for  PervKtnel  inventory  Planning  lender 
ChanfU'-i  Managen^nt  Pobey  ” 14  pp  April  1977. 
(To  be  pTfsentr-l  at  the  NATO  Confereruc  on 
Manpower  Plenmna  and  Orpini/alion  Design.  Siresa. 
Italy.  20  June  I977| 

PH  J>i0 

lioowit/  Stanley  A and  Sherman.  Allan  *'1he 
('haiauenslk's  of  Naval  Personnel  and  PersonnH 
PetfoimarKe,  ' 16  pp  April  1977,  (To  be  pievoied  at 
the  NATO  (onferen^e  un  Manpuwet  Planning  and 
Oigani/aiion  Desig.'i,  Sticaa.  Italy.  20  June  1977) 

PH  HI 

Halut,  Stephen  I and  Stolofl.  Peter.  “An  lr>venlory 
nanmng  MirJel  lot  Nav>  I nlistcd  Pervmnei  ' 35  pp  . 
May  1977,  (Piepaied  for  prescriiali  ft  at  the  Joint 
Natumal  Meeting  of  the  Opetaiiont  Keseaich  StK'iety 
of  America  and  The  Inshtule  fur  Management 
Seteme  9 May  1977,  San  hranuKo.  ( alihfmal 

PH  IH2 

Murray  Rusarll.  2nd.  “Ihe  Ouest  for  the  Perfect 
Study  f Mv  Host  IMA  Daw  at  fNA"  57  pp  . 
April  197? 

PP  183 

Kaaaini.  ilavtd.  '(  hanfrs  in  .Snvirt  Naval  H«>icei  '* 
33  pp  , Nevemher,  1976,  (To  be  (rublished  as  a 
chapter  in  a book  published  hy  TV  National 
5ira(e|ii  Inf'emation  f entcif 

PP  IA5 

Kauing  David.  “New  Technoiofy  and  Naval  Forces 
in  the  South  Atlantic  22  pp  (TEts  iwpei  wai  the 
baats  for  a presentation  made  at  the  Inaiitute  fnr 
f oreigit  PolKy  Analytes.  ( ambridfr.  Mass  , 2S  Apnl 
197?  } 


IV  IK6 

Mi/iaht.  MauiKr  M “PhaK  Spa^e  Iriremali.  With 
out  I muting  Piocedure.  ■ JI  pp  Mav  1977.  (Sub- 
mitted foi  publication  in  Jimmal  of  MaihematKal 
Phvucsl 

PP  t88 

lArnh,  Willtjiri  J . “iritormatiori  PoKCMing  and 
Oiicome  horr^asting  lor  Multilateral  Negotiattons 
Testing  One  AppioaOi"  ^3  pp  , Mav  1977  (Prr 
pared  tor  presentation  to  the  IHth  Annual  Conven 
tion  uf  the  Inteniational  Studies  AiVHiatton.  ( base 
Park  Plaza  Hotel  St  louts.  Miswiuri.  Match  16  20 
1977) 

PP  189 

Code  Russell  t . “Error  Uetection  in  ('ompuleii/ed 
Information  Retrieval  Data  Bases"  July.  1977.  13 
pp  . To  he  pieaenied  at  the  Sixth  Cranfield  Inter 
naiiunal  Conlerencc  on  Mechanized  infotmaiion 
Slurage  and  Kemeval  Systems.  ( ranfietd  Initiluie 
ol  TechrioUvgy.  Cranfleld.  Bediixd,  Hn^and.  26-29 
July  1977 
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